Data - Revision history

GethinWilliams: /* Sort and Search */

2013-11-05T12:38:54Z

Sort and Search

GethinWilliams: /* A Salutary Tale of Copying Files */

2013-11-04T15:10:15Z

A Salutary Tale of Copying Files

GethinWilliams: /* Not Exceeding RAM */

2013-09-10T13:57:31Z

Not Exceeding RAM

GethinWilliams: /* Summary */

2013-08-30T13:37:48Z

Summary

GethinWilliams: /* Rolling Your Own */

2013-08-30T13:36:28Z

Rolling Your Own

GethinWilliams: /* When Data gets Big */

2013-08-16T15:29:57Z

When Data gets Big

GethinWilliams: /* When Data gets Big */

2013-08-16T14:58:49Z

When Data gets Big

GethinWilliams: /* When Data gets Big */

2013-08-16T13:37:10Z

When Data gets Big

GethinWilliams: /* Metadata, Data Curation and Data Re-Use */

2013-08-16T11:18:59Z

Metadata, Data Curation and Data Re-Use

GethinWilliams: /* Meta Data, Data Curation and Data Re-Use */

2013-08-16T11:02:36Z

Meta Data, Data Curation and Data Re-Use

@@ Line 517: / Line 517: @@
 Hash tables (e.g. python dictionaries or sets) can provide very fast O(1) look up times, but come at the cost of some memory overhead (for python, perhaps more than 6x the size of list).
-If you can't avoid the extra space and you'll need to store your data as a list (array).  A linear lookup through an unordered list takes, of course, O(n) time for a list of length n.  If you are going to perform more than one look up (and your data set is large), it will be (well) worth your while sorting the list first.  Using e.g. the quicksort algorithm, the sort will take, on average, O(n log n) time, and using e.g. binary search, the look up will take, on average O(log n) time.  Where 2(log n) << 2n, for large n.
+If you can't afford the extra space and you'll need to store your data as a list (array).  A linear lookup through an unordered list takes, of course, O(n) time for a list of length n.  If you are going to perform more than one look up (and your data set is large), it will be (well) worth your while sorting the list first.  Using e.g. the quicksort algorithm, the sort will take, on average, O(n log n) time, and using e.g. binary search, the look up will take, on average O(log n) time.  Where 2(log n) << 2n, for large n.
 There are many library routines, modules and packages which can provide you with sort and search facilities, so do have a poke about.

@@ Line 72: / Line 72: @@
 |-
 |}
 ===File Formats and File Sizes===

@@ Line 546: / Line 546: @@
 The program will keep running in an infinite loop, randomly accessing cells in a large array, until it is killed.
 Once the program is running, the '''top''' command can give us a useful look at the machine resources and the state of any running processes:
@@ Line 560: / Line 562: @@
 </pre>
-In this case, we can see that the program '''foo.exe''' is showing good CPU utililisation (100%) and that the machine is not using any swap space.
+In this case, we can see that the program '''foo.exe''' is showing good CPU utililisation (100%) and that the machine is not using any swap space.  The total amount of RAM in the machine is shown over to the left: '''Mem:   3341536k total''', which is around 3.3GB.  It therefore makes sense that our program needing '''2GB''' is shown as using close to 60% of the memory of the machine.
 The '''vmstat''' command, where '''si''' shows the amount of data swapped-in to RAM from disk, and '''so''' shows the amount swapped out of RAM onto disk:
@@ Line 578: / Line 580: @@
 Again, we see no swapping activity.
-If we increase N, however, (to 2147483647); recompile and re-run,  we see from top that the memory usage has, of course, increased and the %CPU has plummeted to just 1%!:
+If we increase N, however, (to 2147483647, meaning the program will need '''over 8GB''' of storage space); recompile and re-run,  we see from top that the memory usage has, of course, increased and the '''%CPU has plummeted to just 1%!''':
 <pre>
@@ Line 587: / Line 589: @@
 </pre>
-vmstat tells us a similar tale, where we see activity in the si and so collumns:
+vmstat tells us a similar tale, where we see activity in the si and so columns:
 <pre>
@@ Line 602: / Line 604: @@
 ...
 </pre>
 ===Working on a Shared System===

@@ Line 787: / Line 787: @@
 ** Think about locality of reference.
 ** Go to main memory as infrequently as possible.
 ** Go to disk as infrequently as possible as possible.
 * Check that your are still using the right tools if your data grows.

@@ Line 603: / Line 603: @@
 </pre>
-If you are working on bluecrsytal, you can check the memory & CPU usage of your program by first determining the node(s) that your job is running on, which '''qstat -n <jobid>''', then logging into the node(s) and running the above top and vmstat commands.
+===Working on a Shared System===
 ==Bespoke Applications==

@@ Line 631: / Line 631: @@
 The first sign that you've started to stray into the territory of processing larger data sets may be that you fill up your quota of disk space on, for example, Bluecrystal.  That being the case, you may need to become skilled at shuffling data between some sort of bulk storage pool and the machine doing the work.  '''scp''' can be a very useful tool for this task.
-* [http://en.wikipedia.org/wiki/Map_Reduce Map-Reduce] to process large volumes of data quickly and reliably.
+[http://en.wikipedia.org/wiki/Big_data Big data] is a term used to refer to quantities of data that are hard to deal with using traditional tools.  Since it's definition is relative, it can be used to demark a quantity of data that is at the threshold of mainstream applications.
-* [http://en.wikipedia.org/wiki/Hadoop Hadoop] is a popular, open-source implementation.
 * Seek vs read time.  Seeks can add up, and dominate if files are small. To minimise:
 ** Make use of contiguous access patterns, rather than random ones.
 ** Stream through large files
-* Available on DICE.  Please don't run on bluecrystal
 ==Hadoop & Friends==
 ===Streaming MapReduce===

@@ Line 611: / Line 611: @@
 ==Metadata, Data Curation and Data Re-Use==
-As the name suggests, metadata suggests, it is data about data.  OK, fine.  But why do we want it?  Well, data comes at a price.  There is the cost of creation (e.g. a trip to Antarctica to drill ice-cores) and there is a cost to storing it (e.g. buying servers & disks, paying the electricity bill and the salaries of system administrators).  That being the case, we'd we wise to make the best use of it, once we have it.
+As the name suggests, [http://en.wikipedia.org/wiki/Metadata metadata] suggests, it is data about data.  OK, fine.  But why do we want it?  Well, data comes at a price.  There is the cost of creation (e.g. a trip to Antarctica to drill ice-cores) and there is a cost to storing it (e.g. buying servers & disks, paying the electricity bill and the salaries of system administrators).  That being the case, we'd we wise to make the best use of it, once we have it.  Enter [http://en.wikipedia.org/wiki/Data_curation data curation].  This is the activity that supports the creation of metadata (and the good husbandry of the data in general).  One of the goals of data curation is to make use and re-use of data as easy as possible.
-* http://en.wikipedia.org/wiki/Metadata
+OK, so far so good.  But still a bit abstract for some tastes.  Let's look at an example--a photograph that is tagged with metadata:
-[[Image:Gps-MetadataView.png|450px|thumbnail|centre|Metadata example: time & location of photograph]]
+Breaking this example down we have:
 =When Data gets Big=

@@ Line 609: / Line 609: @@
 If you are using some 3rd party programs to process your data--perhaps supplied to you by a collaborating research group, do come and talk to the folks in the [https://www.acrc.bris.ac.uk/ ACRC], as we may well be able to help you speed up your data processing.
-==Meta Data, Data Curation and Data Re-Use==
+==Metadata, Data Curation and Data Re-Use==
 * http://en.wikipedia.org/wiki/Metadata