Smartedit

'./checkpoints/SmartEdit-7B/Understand-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-7B" --sd_qformer_version "v1.1-7b" --resize_resolution 256 python test/DS_SmartEdit_test.py --is_reasoning_scenes True --model_name_or_path "./checkpoints/vicuna-7b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-7B-v1" --save_dir './checkpoints/SmartEdit-7B/Reason-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-7B" --sd_qformer_version "v1.1-7b" --resize_resolution 256 python test/DS_SmartEdit_test.py --is_understanding_scenes True --model_name_or_path "./checkpoints/vicuna-13b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-13B-v1" --save_dir './checkpoints/SmartEdit-13B/Understand-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-13B" --sd_qformer_version "v1.1-13b" --resize_resolution 256 python test/DS_SmartEdit_test.py --is_reasoning_scenes True --model_name_or_path "./checkpoints/vicuna-13b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-13B-v1" --save_dir './checkpoints/SmartEdit-13B/Reason-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-13B" --sd_qformer_version "v1.1-13b" --resize_resolution 256You can use different resolution to inference on reasoning scenes: python test/DS_SmartEdit_test.py --is_reasoning_scenes True --model_name_or_path "./checkpoints/vicuna-7b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-7B-v1" --save_dir './checkpoints/SmartEdit-7B/Reason-384-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-7B" --sd_qformer_version "v1.1-7b" --resize_resolution 384 python test/DS_SmartEdit_test.py --is_reasoning_scenes True --model_name_or_path "./checkpoints/vicuna-13b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-13B-v1" --save_dir './checkpoints/SmartEdit-13B/Reason-384-15000' --steps 15000 --total_dir "./checkpoints/SmartEdit-13B" --sd_qformer_version "v1.1-13b" --resize_resolution 384Explanation of new tokens:The original vocabulary size of LLaMA-1.1 (both 7B and 13B) is 32000, while LLaVA-1.1 (both 7B and 13B) is 32003, which additionally expands 32000="", 32001="", 32002="". In SmartEdit, we maintain "" and "" in LLaVA and remove "". Besides, we add one special token called "img" for system message to generate image, and 32 tokens to summarize image and text information for conversation system ("..."). Therefore, the original vocabulary size of SmartEdit is 32035, where "img"=32000, ""=32001, ""=32002, and the 32 new tokens are 32003~32034. Only the 32 new tokens are effective embeddings for QFormer.We especially explain the meanings of new embeddings here to eliminate misunderstanding, and there is no need to merge lora after you download SmartEdit checkpoints. If you have download the checkpoints of SmartEdit before 2024.4.28, please only re-download checkpoints in LLM-15000 folder. Besides, when preparing LLaVA checkpoints, you must firstly convert the LLaMA-delta-weight, since it is under policy protection, and LLaVA fine-tunes the whole LLaMA weights.Metrics EvaluationUse the script to compute metrics on Reason-Edit (256x256 resolution): python test/metrics_evaluation.py --edited_image_understanding_dir "./checkpoints/SmartEdit-7B/Understand-15000" --edited_image_reasoning_dir "./checkpoints/SmartEdit-7B/Reason-15000" python test/metrics_evaluation.py --edited_image_understanding_dir "./checkpoints/SmartEdit-13B/Understand-15000" --edited_image_reasoning_dir "./checkpoints/SmartEdit-13B/Reason-15000"Todo List Release checkpoints that could conduct "add" functionality

SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models (CVPR-2024 Highlight)[Paper][Project Page][Demo] 🔥🔥 2024.04. SmartEdit is released!🔥🔥 2024.04. SmartEdit is selected as highlight by CVPR-2024!🔥🔥 2024.02. SmartEdit is accepted by CVPR-2024!If you are interested in our work, please star ⭐ our project.SmartEdit Framework SmartEdit on Understanding Scenarios SmartEdit on Reasoning Scenarios Dependencies and Installation pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url pip install -r requirements.txt git clone cd flash-attention pip install . --no-build-isolation cd ..Training model preparationPlease put the prepared checkpoints in file checkpoints.Prepare Vicuna-1.1-7B/13B checkpoint: please download Vicuna-1.1-7B and Vicuna-1.1-13B in link.Prepare LLaVA-1.1-7B/13B checkpoint: please follow the LLaVA instruction to prepare LLaVA-1.1-7B/13B weights.Prepare InstructDiffusion checkpoint: please download InstructDiffusion(v1-5-pruned-emaonly-adaption-task.ckpt) and the repo in link. Download them first and use python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path "./checkpoints/InstructDiffusion/v1-5-pruned-emaonly-adaption-task.ckpt" --original_config_file "./checkpoints/InstructDiffusion/configs/instruct_diffusion.yaml" --dump_path "./checkpoints/InstructDiffusion_diffusers".Training dataset preparationPlease put the prepared checkpoints in file dataset.Prepare CC12M dataset: InstructPix2Pix and MagicBrush datasets: these two datasets InstructPix2Pix and MagicBrush are prepared in diffusers website. Download them first and use python process_HF.py to process them from "parquet" file to "arrow" file.Prepare RefCOCO, GRefCOCO and COCOStuff datasets: please follow InstructDiffusion to prepare them.Prepare LISA ReasonSeg dataset: please follow LISA to prepare it.Prepare our synthetic editing dataset: please download in link.Stage-1: textual alignment with CC12MUse the script to train: bash scripts/TrainStage1_7b.sh bash scripts/TrainStage1_13b.shThen, use the script to inference: python test/TrainStage1_inference.py --model_name_or_path "./checkpoints/vicuna-7b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-7B-v1" --save_dir './checkpoints/stage1_CC12M_alignment_7b/Results-100000' --pretrain_model "./checkpoints/stage1_CC12M_alignment_7b/embeddings_qformer/checkpoint-150000.bin" --get_orig_out --LLaVA_version "v1.1-7b" python test/TrainStage1_inference.py --model_name_or_path "./checkpoints/vicuna-13b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-13B-v1" --save_dir './checkpoints/stage1_CC12M_alignment_13b/Results-100000' --pretrain_model "./checkpoints/stage1_CC12M_alignment_13b/embeddings_qformer/checkpoint-150000.bin" --get_orig_out --LLaVA_version "v1.1-13b"Stage-2: SmartEdit trainingUse the script to train first: bash scripts/MLLMSD_7b.sh bash scripts/MLLMSD_13b.shThen, use the script to train: bash scripts/SmartEdit_7b.sh bash scripts/SmartEdit_13b.shInferencePlease download SmartEdit-7B and SmartEdit-13B checkpoints and put them in file checkpointsPlease download Reason-Edit evaluation benchmark and put it in file datasetUse the script to inference on understanding and reasoning scenes: python test/DS_SmartEdit_test.py --is_understanding_scenes True --model_name_or_path "./checkpoints/vicuna-7b-v1-1" --LLaVA_model_path "./checkpoints/LLaVA-7B-v1" --save_dir

Two results might be. The paragraph number is only shown for the word and phrase repetition results, though we may change this in the future. Exclusion Lists The first time you use SmartEdit Editor it will check everything in your Document tree, folder or scene — depending on which you selected — unfiltered. But every novel has words and phrases that are part and parcel of the work. In a Harry Potter novel, you wouldn’t be interested in how often Harry or school appeared. You can train SmartEdit Editor to ignore certain results, results that you know are not significant. To do this select a result such as the 92 occurrences of the word waterin the above results list and click the “Add Result to Exclusion List” button on the SmartEdit Editor toolbar. From that point on, no results will be shown in the Repeated Words list for the word water. Every results list has this option and every exclusion list can be edited later to add or remove entries in bulk. The Settings dialog has an option to switch individual exclusion lists on or off, so you can always run a complete check, ignoring the exclusions if you so choose. The screenshot below shows the default entries in the exclusion list for Repeated Words. Common words are included alongside some specific entries I made for the Zane Grey novel I’ve been using, such as Jane, Jerry and Lassiter. It’s worth pointing out that most Editor checks are not case sensitive. The exceptions are the Proper Nouns and Acronyms results which extracts values based on their case. You should now have a basic understanding of how SmartEdit Editor checks and results work and how you work with them. At this point it would be useful to run some of the other checks and examine the varied results they present to you. The image above shows a further list of checks SmartEdit Editor runs independent of the repetition checks. You can run these one at a time by selecting from the menu, or all at once by clicking the Word &