Jinyoon Kim's Webpage

Jinyoon Kim

I am currently pursuing my Master's in Computer Science at the University of Virginia, advised by Professor Yen-Ling Kuo. I received my Bachelor's degree in Computer Science from Penn State University in 2024.

My research interests are at the intersection of Embodied AI, Computer Vision, and Language. I am particularly focused on developing intelligent robotic agents that can seamlessly interact with and assist humans in complex, photorealistic environments. By leveraging foundations in 3D vision and Reinforcement Learning within the Habitat Lab framework, I aim to build agents capable of understanding object affordances and interpreting realistic humanoid actions to facilitate more natural human-robot collaboration.

Previously, I extensively worked on automated medical image analysis and self-supervised learning systems, publishing several papers on uncertainty-aware segmentation and spatially coordinated attention mechanisms. I am now translating these insights into the robotics domain to improve agent reliability and human-robot collaboration.

CV GitHub | LinkedIn

Publications

Affordance-Aware Humanoid Assistance in Photorealistic Embodied AI. On Progress
Building a research framework in Meta's Habitat Lab to model how robots proactively assist humans by interpreting affordance-aware actions. My work involves implementing detailed humanoid-object interactions—where the robot understands how a human uses an object—and uses that context to anticipate and fulfill assistive needs in real-time. Jinyoon Kim. Master's Thesis, University of Virginia, 2026. Advised by Professor Yen-Ling Kuo.
YOLO-SCSA: Enhanced YOLOv8 with Spatially Coordinated Shuffling Attention Mechanisms for Skin Cancer Detection
Jinyoon Kim, Tianjie Chen, Hien Nguyen, and Md Faisal Kabir. In Proceedings of IEEE International Conference on Machine Learning and Applications (ICMLA 2024), pp. 408-415, Dec 15, 2024.
[ paper | code ]
Automated Image Segmentation Using Self-Iterative Training and Self-Supervised Learning with Uncertainty Scores
Jinyoon Kim, Tianjie Chen, and Md Faisal Kabir. In Book of Recent Advances in Deep Learning Applications: New Techniques and Practical Examples, Chapter 1, April 4, 2025.
[ paper | code | Book Weblink ]
Automated Data Labeling for Object Detection via Iterative Instance Segmentation
Jinyoon Kim and Md Faisal Kabir. In Proceedings of IEEE International Conference on Machine Learning and Applications (ICMLA 2023), pp. 845-850, Dec 15, 2023.
[ paper | poster | code ]

Projects

Text-Guided 3D Scene Editing: Volumetric Removal & Generative Addition (2025)
Jinyoon Kim, Sansshita Baskaran, Manvitha Sunireddy [ report | presentation | code ]

Developed an end-to-end pipeline for editing unbounded Mip-NeRF 360 scenes using 3D Gaussian Splatting.
Engineered a custom "Occlusion-Aware Lifting" algorithm to bridge 2D semantics (GroundingDINO + SAM 2) with 3D geometry, enabling precise zero-shot object selection.
Integrated LaMa for multi-view consistent inpainting (removal) and GaussianDreamer for inserting generative 3D assets (addition), solving the "Artichoke Problem" in volumetric editing.

A Reinforcement Learning Pipeline for Financial Reasoning (2025)
Jinyoon Kim, Scarlett Yu, Donggen Li [ report | presentation | code ]

Developed a comprehensive ablation study comparing Deep Contextual RL (PPO, GRPO, RLOO, DPO) against Heuristic Bandits for the FinQA task.
Engineered a "Discriminative Reranking" pipeline to bypass generation failures, successfully training Llama-3.2-3B and TinyLlama-1.1B models.
Demonstrated that RL provides massive gains for weak learners (+35% accuracy on 1B models) while establishing the "SFT Ceiling" effect on capable 3B models.

Capstone Project: Skin Cancer Detection Web Application (2024)
Jinyoon Kim, Tianjie Chen, Hien Nguyen, and Md Faisal Kabir [ slides | project code | web app code ]

Created an accessible and user-friendly web application for skin cancer detection using YOLOv8.
Utilized combined ISIC datasets, implemented confounding factors removal, and incorporated interpretability techniques.
Developed a PWA-based web application ensuring wide accessibility and transparent diagnostics through visual interpretability.

Machine Learning Project: Face Recognition Program (2023)
Jinyoon Kim, Aditya Kendre, et al. [ slides | code ]

Built a face recognition system focused on high accuracy and effective feature extraction.
Developed using a fine-tuned ResNet model and implemented a Top-k features algorithm.
Successfully created a model capable of accurately classifying team members via face recognition.

Plant Village Demo: ML Classification on Mobile Application (2023)
Jinyoon Kim [ code | video ]

Created a mobile application for detecting plant diseases using neural networks.
Developed and fine-tuned MobileNet specifically for plant disease detection on mobile devices.
The resulting application runs efficiently and accurately classifies plant disease images in a mobile environment.

Awards & Honors

National Ackroyd Healthier Days Scholarship, 2024. Awarded for a research project focused on improving health environments for patients through a skin cancer detection application. This project was conducted in collaboration with the Ackroyd Family Foundation and the Penn State community.
PennState Computer Science Department Undergraduate Student Award, 2023. Recognized by the Penn State Computer Science Department for my participation in the ICMLA 2023 conference, where I presented research on a self-supervised learning algorithm for an automated image segmentation system.

News

Pennsylvania State University Capstone Project Conference 2024. May 1, 2024. I attended the Penn State Capstone Project Conference 2024 as a member of the Capstone Project Team. Thanks to Professor Nguyen, Professor Kabir, and my colleague Tianjie Chen.

IEEE International Conference on Machine Learning and Applications (IEEE ICMLA 2023). December 15, 2023. I attended ICMLA 2023 with the poster of my paper for the presentation. Thanks for Dr. Kabir and everyone I met at the conference. [ieee website]