Reproducibility initiative at the Conference on Systems and Machine Learning 2020
[ Back to the conference website ]
Important dates
Paper decision:
January 3, 2019
Artifact submission:
January 15, 2019
Artifact decision: February 10, 2020
Camera-ready paper:
February 17, 2020
Conference:
March 2-4, 2020
Reproducibility discussion session: TBA
Reproducibility chairs
Reproducibility committee
TBA
Process
Following the successful introduction of the artifact evaluation process
at SysML'19
we will continue the reproducibility initiative at the
Conference on Systems and Machine Learning 2020.
The Conference on Systems and Machine Learning promotes reproducibility
of experimental results and encourages code and data sharing
to help the community quickly validate and compare alternative approaches.
Authors of accepted papers are invited to formally describe supporting materials
(code, data, models, workflows, results) using the
standard Artifact Appendix template
and submit it to the Artifact Evaluation process (AE).
Note that this submission is voluntary and will not influence the final decision regarding the papers.
The point is to help authors validate experimental results from their accepted papers by an independent AE Committee
in a collaborative way, and to help readers find articles with available, functional, and validated artifacts!
For example, ACM Digital Library already allows one to find papers with available artifacts and reproducible results!
Artifact submission
You need to prepare your artifacts and appendix using the following
guidelines.
You can then submit your paper with artifact appendix via
dedicated AE website before January 15, 2020.
Your submission will be then reviewed according to the
following guidelines.
Please, do not forget to provide a list of hardware, software, benchmark and data set dependencies
in your artifact abstract - this is essential to find appropriate evaluators!
AE is run by a separate committee whose task is to assess
how submitted artifacts support the work described in accepted papers
based on the standard ACM Artifact Review and Badging policy.
Depending on evaluation results, camera-ready papers will include the artifact appendix and a set of ACM stamps of approval printed on their first page:
At the end of the process we will inform you about how to add badges
to your camera-ready paper.
Public discussion
We plan to organize a public discussion about how to improve the reproducibility of experimental results and reusability of research components at the Conference on Systems and Machine Learning 2020 - please stay tuned for more details.
Questions and feedback