Introducing VerifAI's MultiLLM open-source framework

{
  "Config": {
    "MultiLLM": {
      "rank_callback_file": "example_rank_callback.py",
      "llms": [
        {
          "file": "models/bard.py",
          "class_name": "BARD",
          "model": "chat-bison@001",
          "credentials": "application_default_credentials.json"
        },
        {
          "file": "models/GPT.py",
          "class_name": "GPT",
          "model": "gpt-3.5-turbo",
          "credentials": "/openai/key.json"
        }
      ]
    }
  }
}

 {
  "client_id": "123489-6qr4p.apps.googleusercontent.com",
  "client_secret": "fx-d3456-tryf0g9f9",
  "quota_project_id": "my-llm-training",
  "refresh_token": "1-34GFH89KLwe-eft",
  "type": "authorized_user"
  }

 {
    "organization" : "org-jc8901FDLI0267",
    "api_key" : "rt-067FGDiTL834"
}
  

multillm response Ranking:
1. GPT
2. BARD

Explanation:

• "GPT's implementation of sorting an array is simpler and more concise. It uses the built-in sorted function in Python, which has a time complexity of O(n log n). On the other hand, BARD's implementation uses the merge sort algorithm, which has a time complexity of O(n log n) as well. However, BARD's implementation is longer and more complex, requiring additional helper functions. Therefore, GPT's implementation is ranked higher"

loading module bard...
finished loading module bard
registered model BARD <bard.BARD object at 0x10e4e5b70>
loading module GPT...
finished loading module GPT
registered model GPT <GPT.GPT object at 0x10e4e5d20>
loaded llms: {'bard': <module 'bard' from '/Users/Verifai/models/bard.py'>, 'GPT': <module 'GPT' from '/Users/Verifai/models/GPT.py'>}
calling model: BARD
calling model: GPT
model chat-bison@001

BARD Response:

def newton(f, df, x0, tol=1e-6, maxiter=100):
    """
    Find a root of the function f using Newton's method.

    Args:
        f: The function to find a root of.
        df: The derivative of f.
        x0: The initial guess for the root.
        tol: The tolerance for convergence.
        maxiter: The maximum number of iterations to perform.
        

    Returns:
        The root of f, or None if no root was found.
    """

    for i in range(maxiter):
        x1 = x0 - f(x0) / df(x0)
        if abs(x1 - x0) < tol:
            return x1
        x0 = x1
    return None

GPT Response:

def newton_method(f, f_prime, initial_guess, tol=1e-6, max_iter=100):
    """
    Newton's method for finding the root of a function.

    Parameters:
        f (function): The function for which the root is to be found.
        f_prime (function): The derivative of f.
        initial_guess (float): The initial guess for the root.
        tol (float): The desired tolerance (default 1e-6).
        max_iter (int): The maximum number of iterations (default 100).

    Returns:
        float: The root found by Newton's method, or None if no root is found.
    """
    x = initial_guess
    for _ in range(max_iter):
        fx = f(x)
        if abs(fx) < tol:
            return x
        fpx = f_prime(x)
        if fpx == 0:
            return None
        x -= fx / fpx
    return None

MultiLLM Explanation and Ranking for Example 1:

• "The GPT implementation of the Newton's method function (newton_method) is well-documented with clear parameter explanations and return format. It also includes appropriate input validation checks and handles the case when the derivative is 0. Overall, it is a comprehensive and robust implementation."

• "The BARD implementation of the Newton's method function (newton) is also well-documented and accepts the necessary parameters. However, it lacks input validation checks, such as when the derivative is 0, and does not provide an explicit return value when no root is found. It can be improved by addressing these limitations."

"Based on these factors, I would rank GPT higher than BARD in terms of the clarity, completeness, and robustness of the implementation."

loading module bard...
finished loading module bard
registered model BARD <bard.BARD object at 0x11e881b70>
loading module GPT...
finished loading module GPT
registered model GPT <GPT.GPT object at 0x11e881d20>
loaded llms: {'bard': <module 'bard' from '/Users/frank/Verifai/repo/verifai-ai/src/verifai/llm/multi_llm/models/bard.py'>, 'GPT': <module 'GPT' from '/Users/frank/Verifai/repo/verifai-ai/src/verifai/llm/multi_llm/models/GPT.py'>}
calling model: BARD
calling model: GPT
model chat-bison@001

BARD Response:

def is_prime(n):
  """Returns True if n is prime, False otherwise."""
  if n < 2:
    return False
  for i in range(2, int(n ** 0.5) + 1):
    if n % i == 0:
      return False
  return True

GPT Response:

def is_prime(number):
    if number < 2:
        return False
    for i in range(2, int(number**0.5) + 1):
        if number % i == 0:
            return False
    return True

MultiLLM Explanation and Ranking for Example 2:

• "BARD's implementation of the isprime function is identical to GPT's implementation. Both functions check if a number is prime using the same algorithm. Therefore, both LLMs share the same ranking."

• "GPT's implementation of the isprime function is correct and follows the standard algorithm for checking prime numbers. While the implementation is identical to BARD's, it still demonstrates proficiency in coding logic and knowledge of basic mathematical operations."

loading module bard...
finished loading module bard
registered model BARD <bard.BARD object at 0x1113b1b70>
loading module GPT...
finished loading module GPT
registered model GPT <GPT.GPT object at 0x1113b1d20>
loaded llms: {'bard': <module 'bard' from '/Users/frank/Verifai/repo/verifai-ai/src/verifai/llm/multi_llm/models/bard.py'>, 'GPT': <module 'GPT' from '/Users/frank/Verifai/repo/verifai-ai/src/verifai/llm/multi_llm/models/GPT.py'>}
calling model: BARD
calling model: GPT
model chat-bison@001

BARD Response:

def binary_search(array, target):
    left = 0
    right = len(array) - 1

    while left <= right:
        mid = (left + right) // 2

        if array[mid] == target:
            return mid
        elif array[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return -1

GPT Response:

def binary_sort(arr):
    if len(arr) <= 1:
        return arr
    mid = len(arr) // 2
    left_half = arr[:mid]
    right_half = arr[mid:]

    left_half = binary_sort(left_half)
    right_half = binary_sort(right_half)

    return merge(left_half, right_half)
 
 def merge(left, right):
     merged = []
     i = j = 0
     while i < len(left) and j < len(right):
         if left[i] < right[j]:
             merged.append(left[i])
             i += 1
        else:
             merged.append(right[j])
             j += 1
     while i < len(left):
         merged.append(left[i])
         i += 1
     while j < len(right):
         merged.append(right[j])
         j += 1

     return merged

MultiLLM Explanation and Ranking for Example 3:

• "The GPT solution implements a recursive approach to merge sort. It splits the array into smaller halves, sorts them recursively, and then merges the sorted halves. This approach has a time complexity of O(n log n) and is efficient for large datasets."

• "The BARD solution implements a binary search algorithm. It searches for a target value in a sorted array by repeatedly dividing the search space in half. This algorithm has a time complexity of O(log n) and is efficient for finding a specific element in a sorted array. The implementation is correct and returns the index of the target if found, or -1 if not found. However, it does not involve ranking or sorting other LLMs, which is the task at hand."