我有一个数据框有一系列与作为价值观。 内阵列，1或一笔数目等于一个特定的目标值，我要提取的数值，无论是平等的价值或可以被总结为平等的价值。 我想能够做到这一点，在PySpark.
| Array | Target | NewArray |
| -----------------------|-----------|------------------|
| [0.0001,2.5,3.0,0.0031]| 0.0032 | [0.0001,0.0031] |
| [2.5,1.0,0.5,3.0] | 3.0 | [2.5, 0.5, 3.0] |
| [1.0,1.0,1.5,1.0] | 4.5 | [1.0,1.0,1.5,1.0]|
你可以封装的逻辑作为一个 udf 并创建 NewArray 在此基础上。
我已经借用了逻辑，用于识别要素的阵总结到一个目标值是从 这里.
from pyspark.sql.types import ArrayType, DoubleType
from pyspark.sql.functions import udf
from decimal import Decimal
data = [([0.0001,2.5,3.0,0.0031], 0.0032),
([2.5, 1.0, 0.5, 3.0], 3.0),
([1.0, 1.0, 1.5, 1.0], 4.5),
([], 1.0),
(None, 1.0),
([1.0,2.0], None),]
df = spark.createDataFrame(data, ("Array", "Target", ))
@udf(returnType=ArrayType(DoubleType()))
def find_values_summing_to_target(array, target):
def subset_sum(numbers, target, partial, result):
s = sum(partial)
# check if the partial sum is equals to target
if s == target:
result.extend(partial)
if s >= target:
return # if we reach the number why bother to continue
for i in range(len(numbers)):
n = numbers[i]
remaining = numbers[i+1:]
subset_sum(remaining, target, partial + [n], result)
result = []
if array is not None and target is not None:
array = [Decimal(str(a)) for a in array]
subset_sum(array, Decimal(str(target)), [], result)
result = [float(r) for r in result]
return result
df.withColumn("NewArray", find_values_summing_to_target("Array", "Target")).show(200, False)
+--------------------------+------+--------------------+
|Array |Target|NewArray |
+--------------------------+------+--------------------+
|[1.0E-4, 2.5, 3.0, 0.0031]|0.0032|[1.0E-4, 0.0031] |
|[2.5, 1.0, 0.5, 3.0] |3.0 |[2.5, 0.5, 3.0] |
|[1.0, 1.0, 1.5, 1.0] |4.5 |[1.0, 1.0, 1.5, 1.0]|
|[] |1.0 |[] |
|null |1.0 |[] |
|[1.0, 2.0] |null |[] |
+--------------------------+------+--------------------+
target 列。 为此，我将更新的答案，返回的一个空阵列，如果发生这种情况。 na 或 null 安全。 为精确，我需要一个例子，我想对于较小的范围过6小数位数和它仍然有效。 一个例子将有助于复制。 0.0001 + 0.0031 是 0.0031999999999999997 stackoverflow.com/questions/11950819/python-math-is-wrong/...，我已经更新的答案，以支持精算术来支持你的用例.