<html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div style="" data-md-original="On%2026%2F05%2F14%2016%3A50%2C%20Stephane%20Bortzmeyer%20wrote%3A%3Cbr%3E%26gt%3B%20import%20sys%3Cbr%3E%26gt%3B%3Cbr%3E%26gt%3B%20from%20ripe.atlas.sagan%20import%20PingResult%3Cbr%3E%26gt%3B%3Cbr%3E%26gt%3B%20if%20len(sys.argv)%20%26lt%3B%3D%201%3A%3Cbr%3E%26gt%3B%C2%A0%C2%A0%C2%A0%C2%A0%20raise%20Exception(%22Usage%3A%20%25s%20filename%20...%22%20%25%20sys.argv%5B0%5D)%3Cbr%3E%26gt%3B%3Cbr%3E%26gt%3B%20for%20filename%20in%20sys.argv%5B1%3A%5D%3A%3Cbr%3E%26gt%3B%C2%A0%C2%A0%C2%A0%C2%A0%20results%20%3D%20open(filename).read()%3Cbr%3E%26gt%3B%C2%A0%C2%A0%C2%A0%C2%A0%20result%20%3D%20PingResult(results)%3Cbr%3E%26gt%3B%C2%A0%C2%A0%C2%A0%C2%A0%20print%20filename%3Cbr%3E%26gt%3B%C2%A0%C2%A0%C2%A0%C2%A0%20print%20result.rtt_median%3Cbr%3E%26gt%3B%C2%A0%C2%A0%C2%A0%C2%A0%20print%20%22%22%3Cbr%3E%3Cbr%3EThe%20thing%20to%20remember%20is%20that%20this%20is%20a%20*result*%20parser%2C%20not%20a%20*result**s***%20parser.%C2%A0%20In%20other%20words%2C%20you%20have%20to%20pass%20 each%20result%20individually%20into%20Sagan%20for%20parsing%20and%20it%20will%20in%20turn%20return%20a%20Python%20object%20for%20that%20single%20result.%3Cbr%3E%3Cbr%3EYour%20code%20here%20takes%20*the%20entire%20file*%2C%20multiple%20results%20in%20a%20JSON%20list%2C%20and%20dumps%20it%20into%20Sagan%2C%20which%20explodes%20because%20you're%20passing%20multiple%20results.%C2%A0%20Generally%2C%20this%20is%20bad%20practise%20since%20it's%20entirely%20possible%20that%20your%20one%20file%20could%20be%20bigger%20than%20a%20few%20GB%2C%20which%20would%20definitely%20crash%20your%20system%20if%20you%20tried%20to%20load%20the%20entire%20file%20into%20memory.%3Cbr%3E%3Cbr%3EInstead%2C%20I%20suggest%20the%20following%3A%3Cbr%3E%3Cbr%3E%60%60%60python%3Cbr%3Efor%20filename%20in%20sys.argv%5B1%3A%5D%3A%3Cbr%3E%C2%A0%C2%A0%C2%A0%20with%20open(filename)%20as%20my_results%3A%3Cbr%3E%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%20for%20result%20in%C2%A0json.loads(my_results)%3A%3Cbr%3E%C2%A0%C2%A0%C2%A 0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%20result%20%3D%20Result.get(result)%3Cbr%3E%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%20print(result.rtt_median)%3Cbr%3E%60%60%60%3Cbr%3E%3Cbr%3Eor%20skip%20the%20manual%20JSON%20parsing%20by%20using%20the%20fragmented%20file%20format%20(%60%3Fformat%3Dtxt%60)%3A%3Cbr%3E%3Cbr%3E%0A%60%60%60python%3Cbr%3E%0Afor%20filename%20in%20sys.argv%5B1%3A%5D%3A%3Cbr%3E%0A%C2%A0%C2%A0%C2%A0%20with%20open(filename)%20as%20my_results%3A%3Cbr%3E%0A%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%20for%20result%20in%20my_results.readlines()%3A%3Cbr%3E%0A%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%20result%20%3D%20Result.get(result)%3Cbr%3E%0A%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%C2%A0%20print(result.rtt_median)%3Cbr%3E%0A%60%60%60%3Cbr%3E%0A%3Cbr%3EThe%20key%20is%20to%20remember%20that%20you%20need%20to%20break%20up%20that%20Great%20Big%20File%20into%20separate%20result%20blobs%2C%20either %20by%20looping%20over%20a%20fragmented%20file's%20lines%2C%20or%20by%20parsing%20the%20whole%20JSON%20file%20into%20memory%20first%20and%20parsing%20out%20each%20result.%C2%A0%20That%20step%20is%20up%20to%20you.%C2%A0%20Sagan%20only%20takes%20over%20once%20you've%20got%20a%20single%20result%20to%20work%20with.%3Cbr%3E" class="markdown-here-wrapper" data-md-url="Thunderbird" id="markdown-here-wrapper-60175"> On 26/05/14 16:50, Stephane Bortzmeyer wrote: <blockquote style="margin: 1.2em 0px;border-left: 4px solid rgb(221, 221, 221); padding: 0px 1em; color: rgb(119, 119, 119); quotes: none;"> import sys from ripe.atlas.sagan import PingResult if len(sys.argv) <= 1: raise Exception(“Usage: %s filename …” % sys.argv[0]) for filename in sys.argv[1:]: results = open(filename).read() result = PingResult(results) print filename print result.rtt_median print “” </blockquote> The thing to remember is that this is a result parser, not a results parser. In other words, you have to pass each result individually into Sagan for parsing and it will in turn return a Python object for that single result. Your code here takes the entire file, multiple results in a JSON list, and dumps it into Sagan, which explodes because you’re passing multiple results. Generally, this is bad practise since it’s entirely possible that your one file could be bigger than a few GB, which would definitely crash your system if you tried to load the entire file into memory. Instead, I suggest the following: <pre style="font-family: Consolas,Inconsolata,Courier,monospace;font-size: 1em; line-height: 1.2em;margin: 1.2em 0px;"><code style="font-family: Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px 3px 3px 3px; display: inline;white-space: pre; overflow: auto; border-radius: 3px 3px 3px 3px; border: 1px solid rgb(204, 204, 204); padding: 0.5em 0.7em; display: block ! important;display: block; padding: 0.5em; color: rgb(51, 51, 51);" class="language-python">for filename in sys.argv[1:]: with open(filename) as my_results: for result in json.loads(my_results): result = Result.get(result) print(result.rtt_median) </code></pre> or skip the manual JSON parsing by using the fragmented file format (<code style="font-family: Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px 3px 3px 3px; display: inline;">?format=txt</code>): <pre style="font-family: Consolas,Inconsolata,Courier,monospace;font-size: 1em; line-height: 1.2em;margin: 1.2em 0px;"><code style="font-family: Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px 3px 3px 3px; display: inline;white-space: pre; overflow: auto; border-radius: 3px 3px 3px 3px; border: 1px solid rgb(204, 204, 204); padding: 0.5em 0.7em; display: block ! important;display: block; padding: 0.5em; color: rgb(51, 51, 51);" class="language-python"> for filename in sys.argv[1:]: with open(filename) as my_results: for result in my_results.readlines(): result = Result.get(result) print(result.rtt_median) </code></pre> The key is to remember that you need to break up that Great Big File into separate result blobs, either by looping over a fragmented file’s lines, or by parsing the whole JSON file into memory first and parsing out each result. That step is up to you. Sagan only takes over once you’ve got a single result to work with. </div> </body> </html>