article review – Page 2 – SE metrics (Software Engineering)

Generating documentation from notebooks

https://github.com/jyothivedurada/jyothivedurada.github.io/blob/main/papers/Cell2Doc.pdf

Understanding code is the same regardless if it is in a Jupyter notebook or if it is in another editor. Comments and documentation is the key. I try to teach that to my students and, some of them at least, appreciate it. Here is a paper that can change this to the better without adding to more effort.

This paper introduces a machine learning pipeline that automatically generates documentation for Python code cells in data science notebooks. Here’s a more casual summary of what they did and found:

The Solution – Cell2Doc: The team whipped up a new tool called Cell2Doc. It’s a smart pipeline that breaks down code cells into logical parts and documents each bit separately. This way, it gets more details and can explain complex code better than other tools.
How It Works: Cell2Doc has two main parts. First, a Code Segmentation Model (CoSEG) chops up the code into chunks that make sense on their own. Then, a Code Documentation Model (CoDoc) writes up explanations for each chunk. In the end, you get a full set of docs that covers everything the code is doing.
The Cool Part: This isn’t just about slapping together existing models. Cell2Doc actually makes them better at writing docs for code. It’s like giving a turbo boost to the models so they can catch more details and write clearer explanations.
Testing It Out: They didn’t just build this and hope for the best. They tested it with real data from Kaggle, a place where data scientists hang out and compete. They even made a new dataset for this kind of task because the old ones weren’t cutting it.
The Results: When they put Cell2Doc to the test, it did a bang-up job. It scored way higher on automated tests than other methods, and real humans liked it better too. It was better at being correct, informative, and easy to read.
Sharing Is Caring: They’re not keeping this to themselves. They’ve shared Cell2Doc so anyone can use it to make their code easier to understand.

In a nutshell, Cell2Doc is like a super-smart assistant that takes the headache out of writing docs for your code. It understands the code deeply and explains it in a way that’s easy to get, which is pretty awesome for keeping things clear and making sure your work can be used by others.

I consider to give this tool to my students in the sping when they learn how to program embedded systems in C.

Robustness in language interpretation using LLMs

https://arxiv.org/pdf/2309.10644.pdf

I’ve used language models for a while now. They are capable of many tasks, but one of their main problem is the robustness of the results. The models can produce very different results if we change only a minor detail.

This paper addresses the challenge of interpretability in deep learning models used for source code classification tasks such as functionality classification, authorship attribution, and vulnerability detection. The authors propose a novel method called Robin, which aims to create robust interpreters for deep learning-based code classifiers.

Key points from the paper include:

Problem with Current Interpretability: The authors note that existing methods for interpreting deep learning models are not robust and struggle with out-of-distribution examples. This is a significant issue because practitioners need to trust the model’s predictions, especially in high-security scenarios.
Robin’s Approach: Robin introduces a hybrid structure that combines an interpreter with two approximators. This structure leverages adversarial training and data augmentation to improve the robustness and fidelity of interpretations.
Experimental Results: The paper reports that Robin achieves on average a 6.11% higher fidelity when evaluated on the classifier, 67.22% higher fidelity when evaluated on the approximator, and 15.87 times higher robustness compared to existing interpreters. Additionally, Robin is less affected by out-of-distribution examples.
Contributions: The paper’s contributions are threefold: addressing the out-of-distribution problem, improving interpretation robustness, and empirically evaluating Robin’s effectiveness compared to known post-hoc methods.
Motivating Instance: The authors provide a specific instance of code classification to illustrate the problem inherent to the local interpretation approach, demonstrating the need for a robust interpreter like Robin.
Design of Robin: The paper details the design of Robin, which includes generating perturbed examples, leveraging adversarial training, and using mixup to augment the training set.
Source Code Availability: The source code for Robin has been made publicly available, which can facilitate further research and application by other practitioners.
Paper Organization: The paper is structured to present a motivating instance, describe the design of Robin, present experiments and results, discuss limitations, review related work, and conclude the study.

The authors conclude that Robin is a significant step forward in producing interpretable and robust deep learning models for code classification, which is crucial for their adoption in real-world applications, particularly those requiring high security.

Differential prompting for test case generation

https://arxiv.org/pdf/2304.11686.pdf

Generating test cases is one of the new areas where ChatGPT is gaining traction. It is a good thing as it allows software developers to quickly raise quality of their software.

This paper discusses the problem and challenges in finding failure-inducing test cases, the potential of using LLMs for software engineering tasks, and the limitations of ChatGPT in this context. It also provides insights into how the task of finding a failure-inducing test case can be facilitated if the program’s intention is known, and how ChatGPT’s weakness at recognizing nuances can be leveraged to infer a program’s intention.

The authors propose Differential Prompting as a new paradigm for finding failure-inducing test cases, which involves program intention inference, program generation, and differential testing. The evaluation of this technique on QuixBugs and Codeforces demonstrates its effectiveness, notably outperforming state-of-the-art baselines.

The contributions of the paper include the original study of ChatGPT’s effectiveness in finding failure-inducing test cases, the proposal of the Differential Prompting technique, and the evaluation of this technique on standard benchmarks.

The paper also acknowledges that Differential Prompting works best for simple programs and discusses its potential benefits in software engineering education. Preliminaries and methodology are provided to illustrate the task of finding failure-inducing test cases and the workflow of Differential Prompting.

The authors conclude with the promising application scenarios of Differential Prompting, suggesting that while it is currently best for simple programs, it is a step towards finding failure-inducing test cases for larger software. They also highlight its benefits for software engineering education.

Continuous deployment in systems of systems…

Continuous deployment in software-intensive system-of-systems – ScienceDirect (gu.se)

Interestingly, this is a paper from colleagues of ours from the department. The paper presents how one company – Ericsson – works with continuous deployment of their large software system in 3G RAN (Radio Access Networks). The highlights from the article are as follows:

New software field testing and validation activities become continuous.
Software deployment should be orchestrated between the constituent system.
A pilot customer to partner with is key for success.
Companywide awareness and top management support are important.
Documentation and active monitoring are critical for continuous deployment.

I like this paper because it presents a practical approach and a good set of practices that can be taken up by other companies.

Defect predictions – still valid in 2023…

Industrial applications of software defect prediction using machine learning: A business-driven systematic literature review – ScienceDirect

Wow, when I look at the last entry, it was two months ago. Well, somewhere between the course in embedded systems for my students, delegation to Silicon Valley and all kinds of challenges, the time seemed to pass between my fingers.

Well, nevertheless, I would like to put a highlight to the article from our colleagues who specialize in defect predictions and systematic reviews. The article describes how companies use defect prediction models and when they do it.

It’s a nice sunday reading for those of you who are interested in the topic. It is a good source of best practices as well as a solid source for looking for datasets for defect prediction.

Enjoy your reading!

Creating your own models

Last week I wrote about our seminar and Co-pilot. I’m sure that has stimulated a lot of thoughts on these language models. Many think that this is a difficult task to create, train and use them.

Nothing further from the truth. If you are interested in training such a model from scratch, I recommend the following book (in particular Chapter 4 if you are anxious to get started).

Transformers for Natural Language Processing | Packt (packtpub.com)

The book explains how these models work for natural language processing, but making it work for source code is trivial. Use your code instead of the provided text and there you go. You need a GPU or use some cloud service, otherwise you will wait forever.

But if you have it, you can get really cool results within a day or two.

Good luck!