Dumpsterl User Guide

Each .DAT file contains a regular DETS table that can be directly read by the dumpsterl probe via setting the parameter type to dets and the table to the filename of the .DAT file. The performance gain of skipping Mnesia is usually negligible, as iterating through a DETS table is generally quite slow.

For each ram_copies table, Mnesia keeps a backup in a .DAT file. The table might be written by a running Mnesia at any time (the user can request this via mnesia:dump_tables/1). However, chances are good that the backup was last overwritten with the then-latest table contents last time Mnesia was stopped. So it might not be a good idea to work around Mnesia for a ram_copies table; the on-disk backup might be seriously out of date.

The same kind of .DAT file is present also for disc_only_copies tables. Naturally, this file is always up-to-date with the table contents. Whether or not you want to touch a table file kept open by a running Mnesia instance is an entirely different question. If you have the option to stop Mnesia, we strongly recommend that you do so.

For disc_copies tables, a pair of .DCD and .DCL files are usually found, although the .DCL is not always present. These files are disk_log files (accessible via the Erlang module with the same name, part of the kernel application). The files have different internal formats. A .DCD file holds the contents of a table at a certain point in time as a stream of records corresponding to table rows. The accompanying .DCL contains the “database log” with entries describing changes (inserts, deletes, updates) relative to the snapshot in the .DCD.

Dumpsterl supports directly reading the .DCD, but not the .DCL. Reading the .DCD of a table skips all the overhead of going through Mnesia. It also yields much better probe parallelization (utilizing a larger number of cores, if you have them), because the disk_log is read very efficiently, in chunks that typically contain hundreds of rows. All you need to do is set the parameter type to disk_log and the table to the filename of the .DCD file. Note that in contrast to reading the file backing an opened DETS table, this should always work!

The obvious downside is that the changes already accumulated in the .DCL log will not be taken into account. Simply put, you are not looking at the latest data. In practice, this usually means that the probe will have missed a small percentage of newly inserted records or changes to existing records. The effect is usually negligible if you want to get a general idea of the data in the table, but might be a problem if you are after the latest changes. In that case, there is another option to avoid going through mnesia: set type to ets and read the ets table with the same name.

The following transcript shows setting up and running the probe on the .DCD of our synthetically generated mnesia test table.

The functions ds:spec/3,4 run the probe in a synchronous manner. This means that the caller gets blocked while the probe runs, and receives the spec as return value when the probe is done.

In the above example, we used the Erlang syntax #Record.Field. This is syntactic sugar to refer to the field number of the given field, and requires that the appropriate record definition is loaded into the Erlang shell. This can be done via the Erlang shell built-in rr(Module) where Module is a module that includes the record Record in its source (either directly or via header inclusion).

If we would like to spec the whole record found in the table, we just pass 0 as the field number:

In the above examples, we have not passed any options to the probe. The default setting for limit is a safely low value, 1000 records. Usually, we want to process a much larger amount of data. So we need to raise the limit option setting. In that case it is also a good idea to see progress output and have dumpsterl write dumps of the spec (both periodically as it progresses, and a final one at the end).

We can set the limit to an arbitrary positive integer, or the atom infinity to process all the records in the table. We choose the latter. We also set the progress output to refresh at its default rate, and turn on the dump:

The options passed as the last argument in the above example are dumpsterl probe options with a proplist syntax. For the full list and description of the available options, type help options in the dumpsterl shell, or look for Options: in the reference stored here.

At this point it is worth noting that we received a large Erlang data structure as the return value of the ds:spec/1 function. This is nothing other than the spec itself, the same data that was dumped to the file ds.bin. This spec can be processed further, but the simplest thing we can do with it is pass it directly to a newly opened GUI:

At this point you might become interested in one of the fields in this embedded #ftp{} record. Let’s say that you are after the email field, and not interested in anything else. The way forward is to use a list of chained field specs:

You might have noticed that the probing process got much faster, since the data being spec’d is much simpler. If you have lots of data, it pays to do some preliminary runs on the whole record with a lower limit, and select the field of interest.

You could have accomplished the same result via a field spec function doing the same data selection:

This is usually slightly slower than the equivalent chained field spec, so why would you use it? Obviously because it allows you to do much complex things. Basically, you can perform arbitrary selection of data. You can also perform filtering, by making the fun crash or throw on records that you do not want to include.

You can also combine this with the chained field spec list; items in those list can be any valid field spec. So let’s do something more complex. We will look at the #ftp.password field, but we are only interested in those that are strings shorter than 6 characters. We could do it with a single field spec function, but for the sake of illustrating the above, let’s do it with a chained field spec list.

Looking at the console output and a GUI with the resulting spec, you might have noticed a discrepancy: the probe reported 675,000 records being processed, but in the GUI, the root nonempty_list only has a count of about 450,000. The difference is the number of records where the field spec did not evaluate. The reason in our case is that we deliberately made it crash for data we were not interested in.

Finally, let’s look at an example of specifying attributes. By looking at the data, we have deduced that the field #widget.id contains the table key, and #widget.create_date contains a timestamp when the data originated. In such a case, it pays to specify these as data attributes. Going back to our basic example for the #widget.ftp field, this is how it would look with attributes:

The trick is to pass these attribute specs as a keyed list, included in a tuple with the field spec. Note that for the attributes, we are referencing parts of the record outside the actual data we spec (the #widget.ftp field). This is perfectly fine.

Also note that you can add further attributes (apart from key and ts). Any atom is accepted, and the attribute specs have the same syntax as the field spec. This means you can pass in a fun that receives the whole record and comes up with a value for your custom attribute. In case this fun crashes or throws for a certain record, your attribute will be missing from the attributes of that record.

The functions ds:probe/3,4 take exactly the same options as ds:spec/3,4 that we covered in the previous section. However, by using these functions, the probe is run in a separate process and the call immediately returns with a value that can be used to monitor or stop the probe before completion.

This might be useful if you wish to start multiple probes that run in parallel, or you are not sure that you will wait for the completion of the probe and prefer to have a clean way to shut it down. However, these functions do not yield the spec as their return value, as they return immediately with a {Pid, MonitorRef} tuple. Thus, you will need to read the spec from the dump file to process the result. If you run multiple probes, you also need to arrange for different unique dump filenames.

Please refer to the API documentation of ds:probe/3 for code examples.

To try the following examples, just open a shell with dumpsterl loaded. The quickest way is to type make shell in the dumpsterl toplevel directory to get a fresh, stand-alone Erlang node you can use. Note that the output of the expressions entered in the Erlang shell is not shown below. If you follow along, you will see the raw spec data structure returned at each step.

If you have your values in a list, you can use ds:add/2 directly as the fold function in lists:foldl/3:

Note that when you open a GUI via ds:gui/1, you get the shell back so you can keep on working. You can also open several GUI instances on different data and keep them open. This might be useful if you want to make comparisons.

In case you want to provide attributes with your data, use the add function ds:add/3.

Two specs can be joined with the function ds:join/2. Building on the results of the previous examples:

The information in this joint spec will be the same as if you have added all the terms into a single spec. Adding terms to different specs and joining them later opens the door to Map-Reduce style parallelization of the spec flow. This is in fact exactly how the probe is parallelized.

Up until this point, we have been working with “raw” spec trees. By referring to these specs as raw, we mean that they are in an optimal form for the add and join functions (and the probing process in general) to operate. In particular:

To demonstrate the difference that step 1 means, let’s look at the type definition derived from the joint spec we arrived at above. To do this, we use the function ds:pretty_print/1. In contrast with all the previous examples, here we show the Erlang shell prompt and the output as well:

Let us now post-process the spec via the function ds:postproc/1. This will perform the following transformations on the spec tree:

The pretty-printed result will be a bit different:

It is apparent that the order of subtypes in the toplevel union type has changed. We can also see that all the number() types have been changed to the most specific subtype that fits the data. This is the result of cutting out those abstract types that did not absorb any terms by themselves, and had only one subtype as their child.

What is not so apparent from the above is that the statistics (e.g., the count of terms in each node) have been propagated upwards into the abstract nodes. Only after this step do the parent nodes contain the correct occurrence counts.