Sketches to models…

Image by 127071 from Pixabay

https://www.computer.org/csdl/proceedings-article/models/2023/248000a173/1SOLExN0XaU

It’s been a while since I worked with models and I looked a bit at how things have evolved. As I remember, one of the major problems with modelling was one of its broken promises – simplicity.

The whole idea with modelling was to be able to sketch things, discuss candidate solutions and then to transfer them on paper. However, in practice, this never worked like that – the sheer process to transfer a solution from the whiteboard to a computer took time. Maybe even so much time that it was not really worth the effort of informal sketches.

Now, we have CNNs and all kinds of ML algorithms, so why not use that? This paper studies exactly this.

The paper “SkeMo: Sketch Modeling for Real-Time Model Component Generation” by Alisha Sharma Chapai and Eric J. Rapos, presents an approach for automated and real-time model component generation from sketches. The approach is based on a convolutional neural network which can classify the sketches into model components, which is integrated into a web-based model editor, supporting a touch interface. The tool SkeMo has been validated by both calculating the accuracy of the classifier (the convolutional neural network) and through a user study with human participants. At the moment, the tool supports classes and their properties (including methods and attributes) and relationships between them. The prototype also allows updating models via non-sketch interactions with models. During the evaluation the classifier performed with an average precision of over 97%. The user study indicated the average accuracy of 94%, with the maximum accuracy for six subjects of 100%. This study shows how we can successfully employ machine learning into the process of modeling to make it more natural and agile for the users.

Generating documentation from notebooks

https://github.com/jyothivedurada/jyothivedurada.github.io/blob/main/papers/Cell2Doc.pdf

Understanding code is the same regardless if it is in a Jupyter notebook or if it is in another editor. Comments and documentation is the key. I try to teach that to my students and, some of them at least, appreciate it. Here is a paper that can change this to the better without adding to more effort.

This paper introduces a machine learning pipeline that automatically generates documentation for Python code cells in data science notebooks. Here’s a more casual summary of what they did and found:

  1. The Solution – Cell2Doc: The team whipped up a new tool called Cell2Doc. It’s a smart pipeline that breaks down code cells into logical parts and documents each bit separately. This way, it gets more details and can explain complex code better than other tools.
  2. How It Works: Cell2Doc has two main parts. First, a Code Segmentation Model (CoSEG) chops up the code into chunks that make sense on their own. Then, a Code Documentation Model (CoDoc) writes up explanations for each chunk. In the end, you get a full set of docs that covers everything the code is doing.
  3. The Cool Part: This isn’t just about slapping together existing models. Cell2Doc actually makes them better at writing docs for code. It’s like giving a turbo boost to the models so they can catch more details and write clearer explanations.
  4. Testing It Out: They didn’t just build this and hope for the best. They tested it with real data from Kaggle, a place where data scientists hang out and compete. They even made a new dataset for this kind of task because the old ones weren’t cutting it.
  5. The Results: When they put Cell2Doc to the test, it did a bang-up job. It scored way higher on automated tests than other methods, and real humans liked it better too. It was better at being correct, informative, and easy to read.
  6. Sharing Is Caring: They’re not keeping this to themselves. They’ve shared Cell2Doc so anyone can use it to make their code easier to understand.

In a nutshell, Cell2Doc is like a super-smart assistant that takes the headache out of writing docs for your code. It understands the code deeply and explains it in a way that’s easy to get, which is pretty awesome for keeping things clear and making sure your work can be used by others.

I consider to give this tool to my students in the sping when they learn how to program embedded systems in C.

Robustness in language interpretation using LLMs

https://arxiv.org/pdf/2309.10644.pdf

I’ve used language models for a while now. They are capable of many tasks, but one of their main problem is the robustness of the results. The models can produce very different results if we change only a minor detail.

This paper addresses the challenge of interpretability in deep learning models used for source code classification tasks such as functionality classification, authorship attribution, and vulnerability detection. The authors propose a novel method called Robin, which aims to create robust interpreters for deep learning-based code classifiers.

Key points from the paper include:

  1. Problem with Current Interpretability: The authors note that existing methods for interpreting deep learning models are not robust and struggle with out-of-distribution examples. This is a significant issue because practitioners need to trust the model’s predictions, especially in high-security scenarios.
  2. Robin’s Approach: Robin introduces a hybrid structure that combines an interpreter with two approximators. This structure leverages adversarial training and data augmentation to improve the robustness and fidelity of interpretations.
  3. Experimental Results: The paper reports that Robin achieves on average a 6.11% higher fidelity when evaluated on the classifier, 67.22% higher fidelity when evaluated on the approximator, and 15.87 times higher robustness compared to existing interpreters. Additionally, Robin is less affected by out-of-distribution examples.
  4. Contributions: The paper’s contributions are threefold: addressing the out-of-distribution problem, improving interpretation robustness, and empirically evaluating Robin’s effectiveness compared to known post-hoc methods.
  5. Motivating Instance: The authors provide a specific instance of code classification to illustrate the problem inherent to the local interpretation approach, demonstrating the need for a robust interpreter like Robin.
  6. Design of Robin: The paper details the design of Robin, which includes generating perturbed examples, leveraging adversarial training, and using mixup to augment the training set.
  7. Source Code Availability: The source code for Robin has been made publicly available, which can facilitate further research and application by other practitioners.
  8. Paper Organization: The paper is structured to present a motivating instance, describe the design of Robin, present experiments and results, discuss limitations, review related work, and conclude the study.

The authors conclude that Robin is a significant step forward in producing interpretable and robust deep learning models for code classification, which is crucial for their adoption in real-world applications, particularly those requiring high security.

Differential prompting for test case generation

https://arxiv.org/pdf/2304.11686.pdf

Generating test cases is one of the new areas where ChatGPT is gaining traction. It is a good thing as it allows software developers to quickly raise quality of their software.

This paper discusses the problem and challenges in finding failure-inducing test cases, the potential of using LLMs for software engineering tasks, and the limitations of ChatGPT in this context. It also provides insights into how the task of finding a failure-inducing test case can be facilitated if the program’s intention is known, and how ChatGPT’s weakness at recognizing nuances can be leveraged to infer a program’s intention.

The authors propose Differential Prompting as a new paradigm for finding failure-inducing test cases, which involves program intention inference, program generation, and differential testing. The evaluation of this technique on QuixBugs and Codeforces demonstrates its effectiveness, notably outperforming state-of-the-art baselines.

The contributions of the paper include the original study of ChatGPT’s effectiveness in finding failure-inducing test cases, the proposal of the Differential Prompting technique, and the evaluation of this technique on standard benchmarks.

The paper also acknowledges that Differential Prompting works best for simple programs and discusses its potential benefits in software engineering education. Preliminaries and methodology are provided to illustrate the task of finding failure-inducing test cases and the workflow of Differential Prompting.

The authors conclude with the promising application scenarios of Differential Prompting, suggesting that while it is currently best for simple programs, it is a step towards finding failure-inducing test cases for larger software. They also highlight its benefits for software engineering education.

No rules is also rules…

https://www.amazon.se/No-Rules-Netflix-Culture-Reinvention/dp/0753553635/ref=sr_1_1?crid=1E9EWUHT3W7WV&keywords=no+rules+rule&qid=1694854609&sprefix=no+rules+rule%2Caps%2C387&sr=8-1

Today I had the possibility to read a book a bit outside of what I do today. I used to read a lot of leadership books when I gave my old course in Start-ups. Well, enough of the history. So, I’ve read the book, and it was really nice.

It is a book about modern leadership style from Netflix. It’s written from a perspective of the manager of Netflix (Reed Hastings), but it is commented by a business school professor Erin Mayer (https://erinmeyer.com). It’s a very interesting reading as it provides an account of how leadership of Netflix has evolved over time to what it is today.

Empowerment and flat leadership structure are at the core of this style, but they evolved continuously over years. Candor was the first new leadership style that was introduced and it’s something that all organizations would use. Even universities.

If you are a fan of just Netflix, here is another book that I’ve already reviewed – https://www.amazon.se/That-Will-Never-Work-co-founder/dp/1913068218/ref=sr_1_9?crid=2Q7D6MN6KE682&keywords=netflix&qid=1694855144&sprefix=netflix%2Caps%2C126&sr=8-9 , which has been written from the perspective of one of the founders of Netflix, but not Reed.

Stay safe, stay tuned!

Notes from an apocalypse…

NOTES FROM AN APOCALYPSE : MARK O’CONNELL: Amazon.se: Böcker

I’ve read this book recently as the title and the authors caught my attention. Can you really write notes from the apocalypse? Well, turns out that the authors of this book made a very interesting twist to it.

This book is about people who prepare for the apocalypse. It takes us to a number of places where we meet people who prepare for the worse. For me, the most interesting was a guy who bought an old army bunker and prepared a reasonably priced ranch for survining after a nuclear war. Well, reasonably is still 35,000 dollars, but given that you get to live through the worse, maybe it’s not that expensive.

However, it was not the price that caught my eye. It was essentially how he marketed that shelter. The shelter itself was quite spartan, as opposed to shelter for the ultra-rich people with pools, game rooms, cinemas and what have you.

The main selling point for the shelter was not the spartan condition, it was the dream and the possibility of survival. The owner was selling people on the idea that they will be the ones to create the brave new world after the old one collapses.

I’m not certain that there would be world after the nuclear apocalypse (Chernobyl’s disaster happen 30 years ago and the area will be inhabitable for the next 200 years), but I did like the way he sold the “condos” in the shelter. Quite brilliant, actually.

Human compatible… (book review)

Human Compatible: AI and the Problem of Control : Russell, Stuart: Amazon.se: Böcker

AI is here to stay. We know that. It will only grow in its influence. We know that too. Especially after the release of ChatGPT we know that.

This book looks into different scenarios of co-existence between humans and AI. This is a novel view on the topic, which differentiates this book from the other of this kind. The previous view was either about some sort of doomsday theories how AI takes over the world. Well, there was also a view that AI will never really hit it off, because of the lack of conciousness and a human soul.

This book starts by looking at the historical development of humanity when a new technology was invented. First we have some limitations, which stop us from mass-using this technology. Then, we improve it and start using it a lot, which creates jobs and new markets. Then we automate it so that it can scale fast, which causes mass loss of jobs related to it.

Imagine banking – first, it was manual, which was cumbersome and error prone. Then came calculating machines, which required an army of operators who inputted simple instructions and got results. Then computers came and finally the Internet. Banks are still there, as institutions, but the job of a banker is not the same as 100 years ago. Well, it’s not really the same as 20 years ago; somewhat similar to 10 years ago, but not really.

The same goes with AI and therefore we need to lear how to co-exist with it. We can control it, or we can adjust to it or we can co-develop it and take advantage of it.

I strongly recommend this book as a reading about how to tackle the developments in AI, but more realistically, not doomsday profecy-style.

Continuous deployment in systems of systems…

Continuous deployment in software-intensive system-of-systems – ScienceDirect (gu.se)

Interestingly, this is a paper from colleagues of ours from the department. The paper presents how one company – Ericsson – works with continuous deployment of their large software system in 3G RAN (Radio Access Networks). The highlights from the article are as follows:

  • New software field testing and validation activities become continuous.
  • Software deployment should be orchestrated between the constituent system.
  • A pilot customer to partner with is key for success.
  • Companywide awareness and top management support are important.
  • Documentation and active monitoring are critical for continuous deployment.

I like this paper because it presents a practical approach and a good set of practices that can be taken up by other companies.

Defect predictions – still valid in 2023…

Image by WikiImages from Pixabay

Industrial applications of software defect prediction using machine learning: A business-driven systematic literature review – ScienceDirect

Wow, when I look at the last entry, it was two months ago. Well, somewhere between the course in embedded systems for my students, delegation to Silicon Valley and all kinds of challenges, the time seemed to pass between my fingers.

Well, nevertheless, I would like to put a highlight to the article from our colleagues who specialize in defect predictions and systematic reviews. The article describes how companies use defect prediction models and when they do it.

It’s a nice sunday reading for those of you who are interested in the topic. It is a good source of best practices as well as a solid source for looking for datasets for defect prediction.

Enjoy your reading!

Creating your own models

BIld av Pexels från Pixabay

Last week I wrote about our seminar and Co-pilot. I’m sure that has stimulated a lot of thoughts on these language models. Many think that this is a difficult task to create, train and use them.

Nothing further from the truth. If you are interested in training such a model from scratch, I recommend the following book (in particular Chapter 4 if you are anxious to get started).

Transformers for Natural Language Processing | Packt (packtpub.com)

The book explains how these models work for natural language processing, but making it work for source code is trivial. Use your code instead of the provided text and there you go. You need a GPU or use some cloud service, otherwise you will wait forever.

But if you have it, you can get really cool results within a day or two.

Good luck!