GitHub user JonathanH5 opened a pull request:
https://github.com/apache/incubator-flink/pull/43 Finished Flink-928 Taskmanagers are now displayed in the web interface @rmetzger You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonathanH5/incubator-flink webclient Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-flink/pull/43.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #43 ---- commit 24e8994eac86339ee8b10d0248fd06e5b1ec995c Author: Jonathan <[hidden email]> Date: 2014-06-25T14:47:56Z Finished Flink-928 ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14190720 --- Diff: stratosphere-runtime/src/main/java/eu/stratosphere/nephele/jobmanager/web/ConfigurationServlet.java --- @@ -71,7 +88,37 @@ protected void doGet(HttpServletRequest req, HttpServletResponse resp) PrintWriter w = resp.getWriter(); w.write(obj.toString()); - + + } + + private void writeTaskmanagers(HttpServletResponse resp) throws IOException { + + Set<InstanceConnectionInfo> keys = jobmanager.getInstances().keySet(); + List<InstanceConnectionInfo> list = new ArrayList<InstanceConnectionInfo>(keys); + Collections.sort(list); + + JSONObject obj = new JSONObject(); + JSONArray array = new JSONArray(); + for (InstanceConnectionInfo k : list) { + JSONObject objInner = new JSONObject(); + try { + objInner.put("inetAdress", k.getInetAdress()); + objInner.put("ipcPort", k.ipcPort()); + objInner.put("dataPort", k.dataPort()); + objInner.put("lastReceivedHeartBeat", jobmanager.getInstances().get(k).getLastHeartBeat()); + array.put(objInner); + } catch (JSONException e) { + e.printStackTrace(); --- End diff -- This is not good ;) Can you log the exception as a warn? LOG.warn("Json object creation failed", e); ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14190922 --- Diff: stratosphere-runtime/src/main/java/eu/stratosphere/nephele/jobmanager/web/ConfigurationServlet.java --- @@ -71,7 +88,37 @@ protected void doGet(HttpServletRequest req, HttpServletResponse resp) PrintWriter w = resp.getWriter(); w.write(obj.toString()); - + + } + + private void writeTaskmanagers(HttpServletResponse resp) throws IOException { + + Set<InstanceConnectionInfo> keys = jobmanager.getInstances().keySet(); + List<InstanceConnectionInfo> list = new ArrayList<InstanceConnectionInfo>(keys); + Collections.sort(list); + + JSONObject obj = new JSONObject(); + JSONArray array = new JSONArray(); + for (InstanceConnectionInfo k : list) { + JSONObject objInner = new JSONObject(); + try { + objInner.put("inetAdress", k.getInetAdress()); + objInner.put("ipcPort", k.ipcPort()); + objInner.put("dataPort", k.dataPort()); + objInner.put("lastReceivedHeartBeat", jobmanager.getInstances().get(k).getLastHeartBeat()); + array.put(objInner); + } catch (JSONException e) { + e.printStackTrace(); + } + } + try { + obj.put("taskmanagers", array); + } catch (JSONException e) { + e.printStackTrace(); --- End diff -- Same here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47113486 Thank you. I'll test your code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user JonathanH5 commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47117560 The Taskmanagers are shown in the configuration section (at the bottom) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47120257 Okay, I tried it out. Can you change some stuff? In general, it looks good. The title should be "Task Managers", change "Internet Adress" to "Node", in the "Last Heartbeat" can you show the seconds since the last heartbeat? Is it possible to add a column that shows * the number of slots (Instance.numberOfSlots) * free slots (numslots - allocatedSlots.size() ) * numberOfCPUCores * sizeOfPhysicalMemory * sizeOfFreeMemory I think IPC and Data Port are not that important. Can you move them to the last column? With these changes, the page will be really helpful! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user JonathanH5 commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47126851 Ok, But I will do this after I know what is wrong with https://issues.apache.org/jira/browse/FLINK-978 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user JonathanH5 commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47325361 @rmetzger , ok I added you ideas, can you check it again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14295367 --- Diff: stratosphere-runtime/src/main/java/eu/stratosphere/nephele/jobmanager/web/ConfigurationServlet.java --- @@ -65,13 +90,53 @@ protected void doGet(HttpServletRequest req, HttpServletResponse resp) try { obj.put(k, globalC.getString(k, "")); } catch (JSONException e) { - e.printStackTrace(); + LOG.warn("Json object creation failed", e); } } PrintWriter w = resp.getWriter(); w.write(obj.toString()); - + + } + + private void writeTaskmanagers(HttpServletResponse resp) throws IOException { + + Set<InstanceConnectionInfo> keys = jobmanager.getInstances().keySet(); + List<InstanceConnectionInfo> list = new ArrayList<InstanceConnectionInfo>(keys); + Collections.sort(list); + + JSONObject obj = new JSONObject(); + JSONArray array = new JSONArray(); + for (InstanceConnectionInfo k : list) { + JSONObject objInner = new JSONObject(); + + Instance instance = jobmanager.getInstances().get(k); + long time = new Date().getTime() - instance.getLastHeartBeat(); + + try { + objInner.put("inetAdress", k.getInetAdress()); + objInner.put("ipcPort", k.ipcPort()); + objInner.put("dataPort", k.dataPort()); + objInner.put("timeSinceLastHeartbeat", time); + objInner.put("slotsNumber", instance.getNumberOfSlots()); + objInner.put("freeSlots", instance.getNumberOfAvailableSlots()); + objInner.put("cpuCores", instance.getHardwareDescription().getNumberOfCPUCores()); + objInner.put("physicalMemory", instance.getHardwareDescription().getSizeOfPhysicalMemory() / 1000); --- End diff -- you have to divide by 1048576 here. Dividing by 1024 will turn bytes to kilobytes. The interface shows megabytes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14295464 --- Diff: stratosphere-runtime/src/main/java/eu/stratosphere/nephele/jobmanager/web/ConfigurationServlet.java --- @@ -65,13 +90,53 @@ protected void doGet(HttpServletRequest req, HttpServletResponse resp) try { obj.put(k, globalC.getString(k, "")); } catch (JSONException e) { - e.printStackTrace(); + LOG.warn("Json object creation failed", e); } } PrintWriter w = resp.getWriter(); w.write(obj.toString()); - + + } + + private void writeTaskmanagers(HttpServletResponse resp) throws IOException { + + Set<InstanceConnectionInfo> keys = jobmanager.getInstances().keySet(); + List<InstanceConnectionInfo> list = new ArrayList<InstanceConnectionInfo>(keys); + Collections.sort(list); + + JSONObject obj = new JSONObject(); + JSONArray array = new JSONArray(); + for (InstanceConnectionInfo k : list) { + JSONObject objInner = new JSONObject(); + + Instance instance = jobmanager.getInstances().get(k); + long time = new Date().getTime() - instance.getLastHeartBeat(); + + try { + objInner.put("inetAdress", k.getInetAdress()); + objInner.put("ipcPort", k.ipcPort()); + objInner.put("dataPort", k.dataPort()); + objInner.put("timeSinceLastHeartbeat", time); --- End diff -- can you divide the time by 1000 to make it seconds instead of miliseconds? (Its okay if the interface only says 0 or 1). The "Seconds since last heartbeat" column is mainly useful for detecting machines that are not reporting in fast enough. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14295494 --- Diff: stratosphere-runtime/resources/web-docs-infoserver/js/configuration.js --- @@ -29,4 +29,31 @@ function loadConfigTable(json) { } table += "</table>"; $("#confTable").append(table); -} \ No newline at end of file +} + +/* + * Initializes taskmanagers table + */ +function loadTaskmanagers(json) { + $("#taskmanagerTable").empty(); + var table = "<table class=\"table table-bordered table-hover table-striped\">"; + table += "<tr><th>Node</th><th>Ipc Port</th><th>Data Port</th><th>Time since last Heartbeat</th>" + + "<th>Number of Slots</th><th>Available Slots</th><th>CPU Cores</th><th>Physical Memory (mb)</th><th>Free Memory (mb)</th></tr>"; + for (var i = 0; i < json.taskmanagers.length; i++) { + var tm = json.taskmanagers[i] + table += "<tr><td>"+tm.inetAdress+"</td><td>"+tm.ipcPort+"</td><td>"+tm.dataPort+"</td><td>"+tm.timeSinceLastHeartbeat+"</td>" + + "<td>"+tm.slotsNumber+"</td><td>"+tm.freeSlots+"</td><td>"+tm.cpuCores+"</td><td>"+tm.physicalMemory+"</td><td>"+tm.freeMemory+"</td></tr>"; + } + table += "</table>"; + $("#taskmanagerTable").append(table); +} + +function pollTaskmanagers() { + $.ajax({ url : "configuration?get=taskmanagers", type : "GET", cache: false, success : function(json) { + loadTaskmanagers(json); + }, dataType : "json", + }); + setTimeout(function() { + pollTaskmanagers(); + }, 600); --- End diff -- I would suggest to have a refresh rate of 2000 ms. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14295519 --- Diff: stratosphere-runtime/resources/web-docs-infoserver/js/configuration.js --- @@ -29,4 +29,31 @@ function loadConfigTable(json) { } table += "</table>"; $("#confTable").append(table); -} \ No newline at end of file +} + +/* + * Initializes taskmanagers table + */ +function loadTaskmanagers(json) { + $("#taskmanagerTable").empty(); + var table = "<table class=\"table table-bordered table-hover table-striped\">"; + table += "<tr><th>Node</th><th>Ipc Port</th><th>Data Port</th><th>Time since last Heartbeat</th>" + --- End diff -- Change "Time since last Heartbeat" into "Seconds since last heartbeat" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14295541 --- Diff: stratosphere-runtime/resources/web-docs-infoserver/js/configuration.js --- @@ -29,4 +29,31 @@ function loadConfigTable(json) { } table += "</table>"; $("#confTable").append(table); -} \ No newline at end of file +} + +/* + * Initializes taskmanagers table + */ +function loadTaskmanagers(json) { + $("#taskmanagerTable").empty(); + var table = "<table class=\"table table-bordered table-hover table-striped\">"; + table += "<tr><th>Node</th><th>Ipc Port</th><th>Data Port</th><th>Time since last Heartbeat</th>" + + "<th>Number of Slots</th><th>Available Slots</th><th>CPU Cores</th><th>Physical Memory (mb)</th><th>Free Memory (mb)</th></tr>"; --- End diff -- Change "Free Memory (mb)" into "TaskManager (mb)". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14295897 --- Diff: stratosphere-runtime/resources/web-docs-infoserver/js/configuration.js --- @@ -29,4 +29,31 @@ function loadConfigTable(json) { } table += "</table>"; $("#confTable").append(table); -} \ No newline at end of file +} + +/* + * Initializes taskmanagers table + */ +function loadTaskmanagers(json) { + $("#taskmanagerTable").empty(); + var table = "<table class=\"table table-bordered table-hover table-striped\">"; + table += "<tr><th>Node</th><th>Ipc Port</th><th>Data Port</th><th>Time since last Heartbeat</th>" + + "<th>Number of Slots</th><th>Available Slots</th><th>CPU Cores</th><th>Physical Memory (mb)</th><th>Free Memory (mb)</th></tr>"; + for (var i = 0; i < json.taskmanagers.length; i++) { + var tm = json.taskmanagers[i] + table += "<tr><td>"+tm.inetAdress+"</td><td>"+tm.ipcPort+"</td><td>"+tm.dataPort+"</td><td>"+tm.timeSinceLastHeartbeat+"</td>" + + "<td>"+tm.slotsNumber+"</td><td>"+tm.freeSlots+"</td><td>"+tm.cpuCores+"</td><td>"+tm.physicalMemory+"</td><td>"+tm.freeMemory+"</td></tr>"; + } + table += "</table>"; + $("#taskmanagerTable").append(table); +} + +function pollTaskmanagers() { + $.ajax({ url : "configuration?get=taskmanagers", type : "GET", cache: false, success : function(json) { + loadTaskmanagers(json); + }, dataType : "json", + }); + setTimeout(function() { + pollTaskmanagers(); + }, 600); --- End diff -- I talked to stephan, he suggested to do a refresh rate of 10000ms here (10 seconds). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47354063 Thank you! I really like the page. Can you move the TaskManager's overview into a separate tab / page? (I mean like another page in the menu on the left?) So that you have "Dashboard", "History", "Task Managers", "Configuration"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user JonathanH5 commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14324168 --- Diff: stratosphere-runtime/resources/web-docs-infoserver/js/configuration.js --- @@ -29,4 +29,31 @@ function loadConfigTable(json) { } table += "</table>"; $("#confTable").append(table); -} \ No newline at end of file +} + +/* + * Initializes taskmanagers table + */ +function loadTaskmanagers(json) { + $("#taskmanagerTable").empty(); + var table = "<table class=\"table table-bordered table-hover table-striped\">"; + table += "<tr><th>Node</th><th>Ipc Port</th><th>Data Port</th><th>Time since last Heartbeat</th>" + + "<th>Number of Slots</th><th>Available Slots</th><th>CPU Cores</th><th>Physical Memory (mb)</th><th>Free Memory (mb)</th></tr>"; + for (var i = 0; i < json.taskmanagers.length; i++) { + var tm = json.taskmanagers[i] + table += "<tr><td>"+tm.inetAdress+"</td><td>"+tm.ipcPort+"</td><td>"+tm.dataPort+"</td><td>"+tm.timeSinceLastHeartbeat+"</td>" + + "<td>"+tm.slotsNumber+"</td><td>"+tm.freeSlots+"</td><td>"+tm.cpuCores+"</td><td>"+tm.physicalMemory+"</td><td>"+tm.freeMemory+"</td></tr>"; + } + table += "</table>"; + $("#taskmanagerTable").append(table); +} + +function pollTaskmanagers() { + $.ajax({ url : "configuration?get=taskmanagers", type : "GET", cache: false, success : function(json) { + loadTaskmanagers(json); + }, dataType : "json", + }); + setTimeout(function() { + pollTaskmanagers(); + }, 600); --- End diff -- If we do 10 seconds those heartbeats would be wrong ... . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/43#discussion_r14324245 --- Diff: stratosphere-runtime/resources/web-docs-infoserver/js/configuration.js --- @@ -29,4 +29,31 @@ function loadConfigTable(json) { } table += "</table>"; $("#confTable").append(table); -} \ No newline at end of file +} + +/* + * Initializes taskmanagers table + */ +function loadTaskmanagers(json) { + $("#taskmanagerTable").empty(); + var table = "<table class=\"table table-bordered table-hover table-striped\">"; + table += "<tr><th>Node</th><th>Ipc Port</th><th>Data Port</th><th>Time since last Heartbeat</th>" + + "<th>Number of Slots</th><th>Available Slots</th><th>CPU Cores</th><th>Physical Memory (mb)</th><th>Free Memory (mb)</th></tr>"; + for (var i = 0; i < json.taskmanagers.length; i++) { + var tm = json.taskmanagers[i] + table += "<tr><td>"+tm.inetAdress+"</td><td>"+tm.ipcPort+"</td><td>"+tm.dataPort+"</td><td>"+tm.timeSinceLastHeartbeat+"</td>" + + "<td>"+tm.slotsNumber+"</td><td>"+tm.freeSlots+"</td><td>"+tm.cpuCores+"</td><td>"+tm.physicalMemory+"</td><td>"+tm.freeMemory+"</td></tr>"; + } + table += "</table>"; + $("#taskmanagerTable").append(table); +} + +function pollTaskmanagers() { + $.ajax({ url : "configuration?get=taskmanagers", type : "GET", cache: false, success : function(json) { + loadTaskmanagers(json); + }, dataType : "json", + }); + setTimeout(function() { + pollTaskmanagers(); + }, 600); --- End diff -- Thats fine. We don't want to overload the taskmanager with these requests. Hadoop is also not automatically refreshing the page. If a task manager has not reported in for 30 seconds, the user would still notice. In addition, the user can just reload the page to get an updated version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user JonathanH5 commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47430917 Ok @rmetzger , I finished the changes and in addition to that I simplified how new pages are added to the webInterface. Can you review this? Please test, whether you can still analyze jobs from the history... (and whether the menu works). Just to be sure, but it should be fine :). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47514021 The TaskManagers page looks good, but the History is not working. (JSON parse error). Can you add a testcase to ensure the correctness of the JSON. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
In reply to this post by zentol
Github user JonathanH5 commented on the pull request:
https://github.com/apache/incubator-flink/pull/43#issuecomment-47535896 This was an old bug which was fixed before... . I rebased to the current master and it works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at [hidden email] or file a JIRA ticket with INFRA. --- |
Free forum by Nabble | Edit this page |