MSc Thesis Defense: Abdullah Cevizli, ANDROID MALWARE FAMILY CLASSIFICATION WITH LARGE LANGUAGE MODELS OVER DYNAMIC API CALL SEQUENCES, Date & Time: 21 July, 2026 – 2:00 PM, Place: FENS L065

ANDROID MALWARE FAMILY CLASSIFICATION WITH LARGE LANGUAGE MODELS OVER DYNAMIC API CALL SEQUENCES

Abdullah Cevizli
Computer Science and Engineering, MSc Thesis, 2026

Thesis Jury

Asst. Prof. Orçun Çetin (Thesis Advisor)

Prof. Albert Levi

  Assoc. Prof. Julio Hernandez-Castro

Date & Time: July 21st, 2026 – 02.00 PM

Place: FENS L065

Keywords : Android malware, dynamic analysis, Fine-tuned LLM, JVMTI

instrumentation, open-set recognition

Abstract

Android malware spreads quickly and uses obfuscation to evade static inspection. Dynamic analysis instead observes how an application behaves at runtime and reveals behavior that static analysis misses. Large language models (LLMs) have recently emerged as a powerful tool for security researchers. In Android malware studies, however, they have mostly been applied to static features, and their use on dynamic execution traces remains limited. This thesis studies how accurately malware families can be classified from dynamic API call sequences with a fine-tuned LLM, in both closed-set and open-set settings. The study rests on a collection and segmentation pipeline we built. A custom JVMTI native agent attaches at runtime and records method calls with their argument values, including Binder calls, without recompiling the Android framework. We then segment each trace into behavioral units pivoted on Binder calls. This cuts trace length by a median of 77.9% and brings 86.4% of applications within a 16,384-token budget, up from 48% unfiltered. We fine-tune four small LLMs (LLaMA 3.2 3B, Gemma 3 4B, Qwen3 4B, and Phi-4-mini) with LoRA on the raw segment text. On 22 families the best model, LLaMA 3.2 3B, reaches 97.81% macro F1. The open-set setting adds an OTHER category for families outside the trained set. Here outlier exposure training on a Gemma 3 4B model gives the strongest detection we measured, 92.95% macro F1 at 98.84% AUROC. These results show that fine-tuned LLMs can classify Android malware families directly from the dynamic traces our pipeline produces.