Contents
What does ROS "bag migration" mean?
ROS "bag migration" = the process of converting *.bag recording files so that they contain updated ROS messages even though they were recorded with old message definitions in place.
When you record ROS messages being published to ROS topics, the recording gets saved as a ROS *.bag file. Sometimes, however, you need to update your ROS message definitions via their *.msg definition files. When you do this, you inadvertently "break" the bag file because now the messages it contains in its recording and the message definitions on your file system are out-of-sync. To make the bag file recording usable again, you must "migrate" it. This means simply to convert it from an old set of message definitions to a new set of message definitions so it is still usable and stays in-sync with your latest message definitions in your project. In the bag migration process, bag migration rules, or *.bmr files are created. These act as the interpreting mechanism from the old ROS message definitions to the new ones.
You can create the "bag migration rules" (*.bmr files) using some special rosrun rosbag commands described and demonstrated below, or with `rosbag check`, also demonstrated below. These plugin-based migration rules can be exported by any package, though it is nice to keep the *.bmr bag migration rules files in the same package as the ROS messages (*.msg files) themselves.
Once you have your bag migration rules generated, you can perform the actual bag migration, or, the actual conversion of the *.bag file and the ROS messages it contains, using the `rosbag fix` command.
Here is a summary of the steps involved to perform "bag migration":
Record bag files: You have some *.msg ROS message definition files, and you make some recordings into ROS *.bag files.
Update ROS message definitions: You update your *.msg ROS message definition files according to your needs. Oh no! Your *.bag files are now out-of-sync with your new ROS message definition files, so you need to "migrate your bag files"!
Generate bag migration rules: You generate *.bmr ROS bag migration rule files to be used in this bag migration (bag file conversion) process.
(Optional, but recommended) Export your bag migration rules: You add your bag migration rules (contained in your *.bmr files) to your package.xml ROS package manifest file, thereby "exporting" them for general use by ROS and its various scripts and programs.
Migrate (convert) your bag files (to contain messages according to the new definitions): You "migrate" your bag files by actually converting the bag files from the form with the old message definitions to the form with the new message definitions.
Done! That's what "bag migration" means! That's the full process: update your message files, generate new bag migration rules, convert your bag files to contain the new messages according to the new message definitions. Done.
The steps to generate bag migration rules, export them, and convert your bag files are all explained below.
Bag Migration Rule (*.bmr) Files Overview
Bag Migration Rule (*.bmr) files define one or more Python classes which are subclasses (child classes) of the rosbagmigration.MessageUpdateRule parent class, and which override the appropriate fields and functions. Since they take a specific form, we use the extension .bmr (bag migration rule) in place of the typical .py extension, despite them actually being Python code files. Since they are actually Python code files, however, it is helpful to view them with Python syntax highlighting in your text editor.
The following fields in a *.bmr bag migration rule file must be overridden:
old_type: The full type of the old message.
old_full_text: The full message text.
If necessary, this can be generated from a message with the following command. Note: do NOT include any .msg ending after the input msgname right after savemsg.py. The message name alone, NOT its corresponding file name, is what is required:
rosrun rosbag savemsg.py msgname
You can store the output into a msgname.saved file with:
rosrun rosbag savemsg.py msgname > msgname.saved
new_type: The full type of the new message.
new_full_text: The full message text.
If necessary, this can be generated from a message with the following command. Note: do NOT include any .msg ending after the input msgname right after savemsg.py. The message name alone, NOT its corresponding file name, is what is required:
rosrun rosbag savemsg.py msgname
You can store the output into a msgname.saved file with:
rosrun rosbag savemsg.py msgname > msgname.saved
order: Every message with the same old_type must have a unique order number which contains the order in which they are chained together. Ordering is from lowest to highest order number.
migrated_types: A list of pairs of ("oldsubtype", "newsubtype") which can be used with the migrate helper function.
valid: This must be set to valid = True manually to verify that a human has interacted with the rule file. By changing this value from False to True you are thereby validating that the auto-generated Python update() function defined just below it is in fact correct.
The following method must be overridden:
update(self, old_msg, new_msg): The instance of new_msg passed into this function should be populated from the old_msg.
The bag migration rule `update` function
For the most part just about all of the rule definition will be generated for you by rosbag check or rosrun rosbag makerule.py. The place where the most user intervention is necessary is the Python update function which gets auto-generated for you and placed in the auto-generated *.bmr bag migration rule output file. The update function takes two inputs: old_msg, and new_msg. The intent is to fill in the fields of new_msg appropriately with a combination of default values or values moved over from old_msg.
Here are a few important Python helper classes to make this substantially easier:
self.migrate(old_sub_msg, new_sub_msg) can be used to migrate a sub-message. This migration has the full power of the message migration system behind it and will make use of all available rules (including those defined later in the file).
self.get_new_class(type_name) returns a Python class corresponding to the name of the type passed to it. Important: only instantiate classes using this approach. Never use a message definition imported from a system package as it means the update rule will break in the future if the system definition of the message ever changes.
self.migrate_array(old_sub_msg_array, new_sub_msg_array, type_name) works to migrate an array instead of a sub-message. The extra argument, type_name, is necessary since in most cases; new_sub_msg_array is a empty. type_name will be internally passed through to get_new_class.
Here is an example bag migration *.bmr rule file for a message and its sub-message
Note: With the exception of manually changing the valid member variable in this file from valid = False to valid = True, this entire file was auto-generated by following the directions in the Generating bag migration rules (*.bmr) files section below.
class update_mypkg_MyMsg_cdeceeac6a1a98788dbcb8bf5f31b49b(MessageUpdateRule):
old_type = "mypkg/MyMsg"
old_full_text = """
MySubMsg data
================================================================================
MSG: mypkg/MySubMsg
int32 field1
string field2
"""
new_type = "mypkg/MyMsg"
new_full_text = """
MySubMsg data
MySubMsg more_data
================================================================================
MSG: mypkg/MySubMsg
int32 field1
string field2
float64 another_field
"""
order = 0
migrated_types = [("MySubMsg","MySubMsg")]
valid = True
def update(self, old_msg, new_msg):
self.migrate(old_msg.data, new_msg.data)
#No matching field name in old message
new_msg.more_data = self.get_new_class('MySubMsg')()
class update_mypkg_MySubMsg_4b12e5ff694b0e2a31b2ea9e0bd900f4(MessageUpdateRule):
old_type = "mypkg/MySubMsg"
old_full_text = """
int32 field1
string field2
"""
new_type = "mypkg/MySubMsg"
new_full_text = """
int32 field1
string field2
float64 another_field
"""
order = 0
migrated_types = []
valid = True
def update(self, old_msg, new_msg):
new_msg.field1 = old_msg.field1
new_msg.field2 = old_msg.field2
#No matching field name in old message
new_msg.another_field = 0.
Generating bag migration rules (*.bmr) files
Preferred Approach
Rather than generating *.bmr bag migration rules from bag files after-the-fact, the preferred approach is to generate the necessary *.bmr rules riles at the time that the ROS message definition is first updated. If using Github for version control, for instance, this means we recommend including new *.bmr files in the same Pull Request (PR) in which you change the *.msg ROS message definition files themselves. There are 2 scripts for making this fairly easy to do: savemsg.py and makerule.py.
First, `savemsg.py` allows you to save a full *.msg ROS message definition in a temporary file you specify so that when you then modify your *.msg file you can create a bag migration rule between the original *.msg file (via the temporary file you just created) and the new *.msg file.
Second, `makerule.py` generates a bag migration rule (*.bmr file) to specify conversion rules to convert from your saved instance of the message (created just above with savemsg.py) and the new ROS message currently defined in your system.
TODO: clarify or delete this statement [not sure what this means]: "It is functionally equivalent to creating a recording, except that the file format is human-readable for convenience."
Here is an example of how to use the two scripts above:
In a version-control-system (ex: git) tree at a revision (commit) that contains the old copy of your *.msg ROS message definition file, run the savemsg.py script to back up the ROS message definition file in its original state. Note: do NOT include any .msg ending after the input mypkg/MyMsg right after savemsg.py. The message name alone, NOT its corresponding file name, is what is required:
roscd mypkg rosmake mypkg # use `catkin_make` instead if using the newer catkin build system rosrun rosbag savemsg.py mypkg/MyMsg > MyMsg.saved
Note: What is happening here?
This rosrun rosbag savemsg.py command above first finds the corresponding *.msg ROS message definition file for the ROS message in package mypkg and named MyMsg. This will be the file located at path mypkg/msg/MyMsg.msg. Then, it makes an exact copy of this *.msg file and appends the package and message name as
[mypkg/MyMsg]:
to the top of the file. Lastly, it stores this content, with that one single new line appended to the top, into the new file MyMsg.saved. That's it!
Now, either update your *.msg ROS message file, or change your version-control-system revision to the revision/branch/commit containing the new (updated) *.msg ROS message definition that you previously updated. If using git, you would simply run git checkout new_branch_with_updated_ros_messages, assuming you had previously updated the *.msg ROS message definition file and its changes were stored in that branch or commit.
Next, make sure to rebuild your ROS environment with the new *.msg ROS message definition file. Then, generate a *.bmr bag migration rule file with the makerule.py script. Both of these steps are shown here. Note that if you have moved the *.msg ROS message definition file since you first ran the savemsg.py script above, it will prompt you for the new location.
rosmake mypkg # use `catkin_make` instead if using the newer catkin build system rosrun rosbag makerule.py MyMsg.saved myrule.bmr
At this point, you will need to edit myrule.bmr file by hand. See the Rule Files section previously described above for more info.
Alternative Approach
The rosbag check program can also be used to populate most of the rule templates for you so long as the message type names still match.
To generate rules:
rosbag check in.bag -g myrules.bmr
Note: Using this functionality is generally NOT advised for creating permanent rules to check into the system. Instead, follow the preferred approach above. Creating rules directly from arbitrarily old bag files is likely to skip intermediate changes and lead to message ordering confusion. However, if a previously-recorded *.bag file is missing a corresponding set of bag migration rules to allow you to migrate (convert) it, this is intended to be the quickest pathway to making the bag file functional with your new ROS message definitions.
At this point, you will need to edit myrule.bmr file by hand. See the Rule Files section previously described above for more info.
To make sure your bag migration rules in your generated *.bmr file are sufficient, you can now pass the *.bmr rule file into rosbag check explicitly, like this:
rosbag check in.bag myrules.bmr
Exporting your bag migration rules (*.bmr) files
Finally, if you want your bag migration rules (*.bmr files) from above to be permanently exported for people to use, add the following export to your package.xml ROS package manifest file:
<export> <rosbag migration_rule_file="myrule.bmr"/> </export>
This will cause the bag migration rule contained in your myrule.brm file you just created above to be automatically loaded whenever the ROS message migrator is instantiated.
NOTE: in order for a bag migration rule "plugin" to be exported correctly, you must depend on the package the plugin is used for. In this case your ROS package MUST have a dependency on rosbag to export migration rules. This is done by also adding the following to your package.xml ROS package manifest file:
<depend package="rosbag"/>
<run_depend>rosbag</run_depend>
As of ROS Groovy you can also avoid the dependency on rosbag and instead depend on the new package called rosbag_migration_rule, like this:
<depend>rosbag_migration_rule</depend>
If you do so, the XML export tag must be named like this. Notice that rosbag migration_rule_file="myrule.bmr" is now written as rosbag_migration_rule rule_file="myrule.bmr":
<export> <rosbag_migration_rule rule_file="myrule.bmr"/> </export>
The end result should now look like this. Notice that 3 separate bag migration rule files are shown exported here just as an additional demonstration:
<depend>rosbag_migration_rule</depend> <export> <rosbag_migration_rule rule_file="myrule1.bmr"/> <rosbag_migration_rule rule_file="myrule2.bmr"/> <rosbag_migration_rule rule_file="myrule3.bmr"/> </export>
Migrating (converting) your *.bag files
If you have NOT exported your bag migration rules
Now that you have created bag migration rules you can use them to migrate (convert) your old *.bag files to thereby convert the old ROS messages they contain into the new ROS messages which you have defined, by running the following `rosbag fix` command. Notice that here you are explicitly specifying which bag migration rules *.bmr file to use:
rosbag fix myfile.bag myfile_migrated.bag myrules.bmr
If you HAVE exported your bag migration rules
Notice that in the command above you are explicitly specifying which bag migration rules *.bmr file to use in the bag migration (conversion) process. If you have also exported your bag migration rules *.bmr file via your package.xml ROS package manifest file, per the exporting instructions above, then you can instead run this next command. In this case, we first make the ROS packages in order to incorporate the bag migration rules you exported above, as well as the defined ROS messages, into your workspace, then we run the rosbag fix command to convert the bag file:
rosmake mypkg rosbag fix myfile.bag myfile_migrated.bag
catkin_make rosbag fix myfile.bag myfile_migrated.bag
It is also fairly common to use the --smart --force options, like this:
rosmake mypkg rosbag fix --smart --force myfile.bag myfile_migrated.bag
catkin_make rosbag fix --smart --force myfile.bag myfile_migrated.bag
You can see what they do by running the rosbag fix --help command, as shown here:
$ rosbag fix --help
Usage: rosbag fix INBAG OUTBAG [EXTRARULES1 EXTRARULES2 ...]
Repair the messages in a bag file so that it can be played in the current
system.
Options:
-h, --help show this help message and exit
-n, --noplugins do not load rulefiles via plugins
--force proceed with migrations, even if not all rules defined
--smart Try converting even if the bmr is missing by copying fields
that have matching names.See the `rosbag fix` command-line page for more information.
END