Alpaca-LoRA Chatbot Project with Excellent Features

This project is still under development, but it already offers amazing features that users will love. While working on the project, I noticed some errors that the model would generate, such as too many line breaks, which ultimately led to OOM. However, I always merge any improvements that I discover after finding any issues. This repository showcases Alpaca-LoRA as a chatbot service, complete with Alpaca-LoRA and Gradio. All you have to do is open the alpaca_lora_in_colab notebook in Colab and run each cell in order to use the project. With a standard GPU instance (T4), you can run the 7B and 13B models. With a premium GPU instance (A100 40GB), you can even run the 30B model! The project offers two modes: batch generation and streaming. Users can request to generate prompts in bulk with batch generation, which aggregates requests and waits for them to finish. For example, if a user submits a request for a batch_size of 4, that request will be processed while additional users can still submit their own requests. The streaming mode, on the other hand, processes requests one at a time as they come in.